Fast Training Convergence Using Mixtures of Hidden Markov Models

In this paper, we describe a method for mixing an arbitrary number of discrete fixed-structure Hidden Markov Models such that we retain their collective training experience. We show that, when presented with a novel data sequence, this new mixture model converges in significantly fewer iterations than either a randomly initialized model or any one of the mixture’s component models. We also explore the notion of long-term memory to further exploit the component models’ collective training experience. Several general approaches to integrating memory into the parameter updates are considered, including decaying the expectations derived in the E step of online-EM, and windowing techniques where we use a variable sized data sample. Our experiments show that using mixtures of HMMs as initial parameter settings can speed convergence significantly. In some cases, the mixture models required as few as 30% of the iterations required for a randomly initialized model. Additionally, we find that mixture models show significant improvements in the ability to adapt to data produced by non-stationary distributions.